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Some aspects of a grass roots evaluation training 
program are presented. The program consists of tw elements;: (1) a 
series of 11 slide/tape individualized self -paced units, and (2) a 
six-week summer program. Three points of view on this program are: 
(1) University graduate programs in quantitative areas are usually 
consumed by specialists; (2) Professionals are frequently being 
required to make critical evaluation decisions; and (3) A large 
proportion of the decisions made at the building level require 
practical training. A unit outline of the program is attached. 
(CK) 
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I have slightly changed the title and expanded the 
scope of this paper from that which you were led to 
expect* I did this because I kind of wanted to, and 
because I felt the penalties for co doing probably were 
not very severe* 

In any event , in the next very few minutes, I want 
to describe some aspects of a grass roots evaluation 
training program which is being bom, albeit with some 
difficulty, at the School of Education of Northwestern 
University. In the process, I hope to introduce my 
philosophy about the need for such a program, in the 
hopes that it might enter into the deliberations of other 
institutions. 

At this point, the actually developed program consists 
of two major elements: 

1) A series of eleven slide/tape, totally indivi- 
dualized cind self-paced units dealing with important topics 
in the world of project evaluation. The overall goal, as 
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stated to the Wieboldt Foundation which originally funded 
the effort, was to develop a technique for people on pro- 
jects — people untrained in evaluation methods — to 
train themselves to handle some of the frequently occurring 
evaluation tasks on a project. 

2) The second aspect of our "grass roots" training 
program is a six-week summer program directed toward 
tooling-up in-service professionals to handle routine 
evaluation tasks. These professionals would not have 
career goals in the direction of evaluation, but would 
be people like assistant superintendents, pri-ncipals, 
counselors, or interested teachers who would stay in 
their regular position, but would be prepared to hamdle 
frequently recurring evaluation tasks at the school 
building level. 

So . . • the two currently operative aspects of c ir 
program involve some self-instructional tinits and in- 
service training to prepare practicing educators to solve 
local problems. What motivates such a program? Three 
interlocking points of view come to mind. 

First, university graduate programs in quantitative 
areas are usually being consumed by specialists or 
specialists in training. Most of these people are working 
toward Ph.D. degrees. Few of these graduates will ever 
end up at tlie school building levels helping the practicing 
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educators make day-to-day evaluation decisions. 

Second, many well meaning professionals at 
the building level are frequently being required to make 
evaluation decisions which are critical. These kinds 
of people have continually expressed to me their lack 
of confidence in these types of areas. 

Finally, we observed that a large proportion of the 
decisions which must be made at the building level are 
not all that technical or theoretically that difficult. 
That is, a Ph.D. or even a Masters in evaluation techniques 
is really not required, what is required is some very 
specific, practical training program directed toward these 
practical evaluation problems. 

With this background in mind, let me describe the 
two elements of the grass roots training program which 
are operational at this time. 

First, the self -instructional units: These eleven 
units were developed over about an 18-month period and 
have gone through at least two sets of revisions. They 
are currently being produced and distributed through the 
Center for the Teaching Professions. The operation of 
the units goes like this: 

First, the potential user needs to have a problem 
which is basically an evaluation problem. For example, 
perhaps he is a doctoral student who needs to construct 
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a questionnaire; or a building principal who needs to 
present the data from a survey in some sort of a printed 
report; or a teacher who wants to choose a standardized 
test for some reason • We have found , in our pilot studies 
and in the data gathered from people who have been using 
the units, that if a person uses the unit because a 
real need exists, the feedback is very good. In cases 
where people have used the units for large group instruction, 
the feedback has been less satisfactory. We believe that 
the reason for the dissatisfaction is frequently because 
the user feels he is receiving information which is of 
little importance to him or her. 

Next, given that a particular subject has been 
identified for which a unit is prepared, the user reads 
a brief advance orgcinizer which covers the major topics 
in the unit. This takes no more than five minutes. 

The third step is the actual individualized presentation 
of the unit. The unit consists of approximately 60 slides 
with a 60-minute tape. The tape directs the user through a 
printed heindout which is coordinated with the slides. The 
picture and the audio part may direct the s undent to do a 
particular example or solve a problem before moving on to 
subsequent slides. Thus, we have built into the system 
two techniques for active involvement by the user: First, 
the user must do these problems and answer the questions 
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which are part of the unit. Second, the user must actually 
handle the advancing of the slides manually. We did this 
on purpose So that the user would not be a passive receiver 
of i.i formation, but would rather be forced to take part 
in tl.e learning process. Finally, after the person has 
comp:eted the units and reviewed any notes he may have 
taken, a mastery test is available. Some people prefer 
not to take the test and feel that it is unnecessary — which 
is fine; but in instructional settings, it is frequently 
desireable to have some sort of performance measure. 

The entire process involves somewhere from an hour 
and twenty minutes to two hours — depending on how many 
times the user stops the tape and the slides to take notes 
and review questions. 

We have a handout which lists the topics of the units 
and gives a brief description of each. Briefly, though, the 
units cover topics like: choosing a sample of different 
kinds, testing statistical hypotheses, avoiding common errors 
in evaluation and research, constructing questionnaires, 
selecting and using standardized tests, some technical issues 
in testing, interviewing techniques, data presentation tech- 
niques, computer applications in education, some hints for 
project evaluators, and specifying behavioral objectives. 

These units have been used in a variety of setting?. For 
example, funding agencies have purchased some of the units 

ERIC 



to be used for in-service programs to train staff members 
for better evaluation cf proposals. Universities and 
colleges have the units on file to help with instruction 
and research • They have been used by school districts for 
in-service work. It appears to us, given the very positive 
response we have received from people who have viewed the 
units, that this kind of approach to grass roots evaluation 
is well-received. 

The second aspect of our program involves the in- 
service training of practicing educators to handle key 
evaluation problems. The program has not been conceived 
as a degree program, although degree credit can be earned. 
The program has also not been conceived for graduate 
students in training. We are trying to limit enrollment 
to people who are actually working in school buildings and 
who intend to return to that same building in the following 
year. The training program will involve a good deal of case 
study work. 

A participant will have the option of enrolling in 
an on-going evaluation practicum during the eight-week 
experience. That is, the participant can enroll in an 
independent study under the direction of a faculty member 
where the topic of study would be a current evaluation 
problem in the district. The instructional techniques will 
include self-instructional units, simulation activities.. 



books and lectures. 

The workshop will involve six or eight weeks (depending 
on the enrollment option chosen) of intensive training in 
sepcific evaluation topics. in addition, a series of district- 
wide in-service programs on evaluation topics will be out- 
lined for possible use in the district. 

For students enrolled in the Graduate School, two or 
three units of graduate credit can be earned, depending 
on the eirollment option chosen. A "course" is one academic 
quarter >f work and is the equivalent of two and two-thirds 
(2 2/3) semester hours or four quarter hours of credit. Only 
students who are enrolled in a degree program in the Graduate 
School at Northwestern will receive credit through the 
Graduate School. Other students will receive either under- 
graduate credit or non-degree graduate credit. 



Topic : 



SELF-INSTRUCTIONAL UNITS ON EVALUATION AND RESEARCH 



Project Director: John W, Wick 

Associate Professor of Education 



Direct Inquiries to: The Center for the Teaching Professions 

2003 Sheridan Road 
Northwestern University 
School of Education 
Evcuiston , Illinois 60201 



This document will have four sections ^ as follows: 

I. Brief outline of t:he eleven self -instructional units, 

II. Description of the format and technique used in the units, 
III. Brief list of possible uses for the units. 

IV. Description of other related materials which will 
soon be availcdDle. 



I. BRIEF OUTLINE OF THE ELEVEN SELF- INSTRUCTIONAL UNITS 



UNIT I, "Overview," introduces the entire program and covers 
several data collection plcuis, plus the question of sampling. 
Topics include: 

survey vs. sample survey — survey vs. questionnaire — 
justifying a survey — mail surveys — population — types 
of data — types of surveys — types of sampling designs — 
levels of content analysis; cautions — unique characteristics, 
sources, and bias in historical research — types and problems 
of developmental studies. 
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UNITII, ^'Experiments: Testing Statistical Hypotheses," concerns 
itself with the experiment itself, the results, and sources of 
error. Topics include: 

stating the general problem — limiting the problem — defining 
measures — stating statistical and research hypotheses — 
perfoxnning the experiment — probability distribution — comparing 
the results to the statistical hypothesis — making a conclusion - 
type I error — type II error — level of significance. 

i 

UNIT III , •'Avoiding Common Errors in Evaluation and Research," 
begins with a review of the hints and cautions included in the 
other ten units, and continues with cautions in the following 
four areas of experimental design: 

defining and controlling variables — choosing a sas^>le and 
making inferences from it — constructing artificial stimulus 
and response situations in the assessment phase — providing 
for reliable interpretation of results. 



UMIT IV , "Questionnaire Construction," presents basic guidelines 
for preparing a questionnaire from deciding whether to use one 
through plemning follow-up procedures. Topics include: 

Is the questionnaire the best technique? — advantages and 
disadvantages — specific steps to follow, from defining 
population to preparing report — appearance information 
to include — directions — cover letter — mailing — follow-up 
efforts — techniques for non-responders — decisions about 
content and wording — item fomats — special scaling formats. 



UI^IT V , "Selecting and Using Standardized Tests" is an introduction 
to the classification of standardized tests and types of item 
formats. Topics include: 

How to find a measure you need — differences between standardized 
and nonstandardized tests — common classification of standardized 
measures — maximum performance tests classified by use — typical 
performance tests classified by use — cautions in typical per- 
formance tests — types of item formats — cautions in choosing 
item formats. 
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UNIT VI ^ "Technical Issues in Testing considers standards of 
cooqparison for tests and matters of test accuracy. Topics include: 

norm- referenced measures (method of construction, key ideas, test 
items, uses reporting scores) — criterion referenced measures 
(same information) — validity — reliability — standard error 
of measurement — cultural bias. 



UNIT VII , "Data Collection by Interview" introduces the characteristics 
of the interview as a data gathering technique, and presents 5tep- 
by-step procedures. Topics include: 

characteristics — potential uses — prior considerations — 
defining your research problem — specifying the objectives o ' 
the study — considerations in the design of the study — chousing 
the saii^>le — type of qxiestions to ask — wording and question 
order — importance of setting the atmosphere — initial impr ssion 
guidelines for questioning — recording responses — using pr bes — 
closing the interview — tabulation and analysis of results - 
errors caused by interviewer's personality and approach — re ording 
errors — noa-respondonts • 



UNIT VIII , "Data Presentation Techniques" deals with various 
kinds of charts, graphs, and tables that are frequently used to 
present data in final reports. Topics included: 

ordered vs. unordered categories — frequency distribution — 
single vertical or horizontal bar graphs — frequency polygon — 
cumulative display (ogive) — compound bar graph — component bar 
graph — bilateral bar graph — "pieces of pie" display — some 
things to avoid in charts — scatter diagrams (data trend charts) — 
rules for creating tables — purpose of tables — gathering and 
interpreting data for tables — cautions. 



UNIT IX , "A Layman's Introduction to Computer Applications," orients 
the neophyte to the computer including how it works and how it can 
help him. Topics included: 

computer characteristics — data processing vs. confuting vs. 
computer — input, handling, and output processes — related 
machines for the computer user — steps in writing a program — 
computer storage and tabulation of survey results and tests — 
con^uter simulation — computer assisted instruction and con^uter 
managed instruction — national data barnk. 
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UNIT ""Hints for Project Bvaluators** is designed as a guide for 
project directors and project evaluators. Topics included: 

impact of the program — project "prosdses"' — budget — prior 
commitments and real poirer — timetable — continuous outcome vs. 
intervention evaluation — considerations in a continuous assess- 
ment i\odel — the evaluator as a cosmunicetions facilitator — 
••action now" philosophy — the evaluator as number interpreter — 
evaluating objectives. 



UNIT q^** Specifying Beliavioral Objectives" teaches the user how 
to wr wte and evaluate behavioral objectives. Topics included: 

perfo nnance tsrms — conditions of perf orv uice — criterion 
level — cov^nitive levels (taxonomy) — v€ rb choice — why use 
behav ioral ODjectives — how to use them « - when to use them — 
where to use them — answering criticisms >f behavioral objectives. 



II. DESCRIPTION OF FORMAT AND TECBMIQCOB JSED IN THE UNITS 



The project format is designed for i];dividual umb with 
maximum flexibility. The one or more 1 idi.vidual work positions 
which would be established require only a manually operated slide 
projector, a cassette tape player (probably equipped with headsets) , 
and a place for the student to %rork« To maxiAixe the individuality 
of the units # the tape recorder and slide projector operate in- 
dependently of each other, to allow the student to progress through 
the unit at his own pace. That is, he can listen to the recorded 
message as often as necessary, stop to take notes, or review a 
series of slides more than once. 

Individualized flexibility is also created by the self- 
contained character of each unit. That is, the user can choose 
to use only those vmits which are most appropriate to his needs. 
While the units are indeed closely related, each operates in- 
dependent of the others and focuses on a specific concept. Thus 
it is not imperative that the user coflq[>lete all eleven units to 
benefit frcxn the program « 

The four steps in the presentatidn procese are Pre-organization, 
Instruction, Evaluation, and Feedback. Each is described in more 
detail below. 
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Stap 1; Pre-Qrfyan<zation 



The person is expected to read 
a very brief (perhaps 6 to 8 page) 
outline of the unit before beginning 
to interact with the machine. This 
pre-reading operates as an advance 
organizer for the user. Most people 
are more comfortable and confident 
in working through a unit if they 
know in advance the general areas 
of content which will be covered. 



UNIT 
ORIENTATION 

(objectives, 
handout, over- 
view) 

Cong>onent 1 



T 



Read handout 
overview. 



Read objec- 
tives • 



Step 1 



The pre-reading actually has 
another pui^pose as wll. The 

h^?T^urS r*-!;*''^.**^^ self -instructional units leads us to 

belxeye that the student must be actively involved — in a ohvsical 

tlvlrl^ell ^STstudLrL-: presentation. If 'So^^aJ^enSon 

wavers and the student begins to view the slides as if he were 

Jo^ai;;?^%!lT!;^ commentary or dramatic presentation on television. 

H^i2 questions are asked in the actual presentation, and 

the handout leaves spaces for responses to these questions. The 

tr?Je°??„S^«/;U^ ^"^'^ ^ ^'^y ^^"^ responl^ is tS^^rtant 
to the flow of the program so that the student will be at a dis- 
advantage if he simply does not respond. 



Step 2; Instruction 
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The primary instruction takes 
place in this step. The instruction 
cones through three modes. 

VISUAL: Pictures, charts, 
diagrams, outlines, cartoons, a 
variety of visual forms cire pre- 
sented through the use of a slide 
projector. 

AUDIO: Linked to the visual 
part is the audio presentation, 
through the use of cassette-type 
recorder and playback equipnent. 



SLIDES 



CASSETTE 
TAPE 



PRINTED 
QUESTIONS 
ON HANDOUT 

Component 2 

T 



Set up tape 
recorder and 
slide projector. 



Begin tape and 
slides , stopping 
when you wish 
and filling in 
handout questions 
as they come up. 

Step 2 



PRIMTBD INFORMATION AND WRITTEN RESPONSES: As 
m Step 1, the pre-reading handout leaves room for 
questions asked by the visual or audio parts. This 
allows room for the students to take notes. 

imporSnt^^^^ comments regarding step 2 are appropriate and 



mentioned 
responses to 
handout also 
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a. Equipnent can be purchased which synchronizes the 
audio and viOmo parts of the presentation. That is, through 
the use of electronic signals, the slides will advance when 
a certain portion of the audio has been ccnpleted. in these 
units, the synchronised equipnent will not be used. Three 
inportant reasons exist for this decisionT In the first place, 
when the student must advance each slide individually by 
himself it keeps him alert to the presentation. That is, 
advancing to the next slide is another advance organizer. 
Secondly, freedom from a prograimned time sequence allows for 
individual differences. Some students will want to dwell on 
certain slides, or take more detailed notes than others. 
Finally, the initial cost and upkeep on the synchronized units 
are considerably higher. 

b. Three modes are used (audio, visual, printed) and 
all three modes are used in the primary instruction. That 
is, sometimes the audio is the prime instructor, backed up 
by the visual and printed. Other times one of the other two 
modes takes over as the prime instructional mode. This is 
done to make the presentaticms more interesting and to keep 
the student attentive to all three modes. 



St^ 3; Evaluation 



When the student has completed 
the machine interaction to his own 
satisfaction and has reviewed his 
notes for a few minutes, a mastery- 
type test is administered. The 
mastery test will be criterion 
referenced. That is, the specific 
objectives of the unit will be stated 
for the student, and the items will 
reflect mastery or non-4Bastery on 
these objectives. The student's 
performance on the mastery test 
will not be compared to other students. Instead, it is compared 
to the criteria specified. Only two scores are reported: either 
the student reaches mastery on the criteria, or he does not. No 
grades, percentiles, standard scores, or stanines are reported. 



MASTERY 
TEST 

(criterion 
referenced to 
objectives) 

Con^nent 3 



J 



Review notes, 
handout , and 
objectives* 



Take mastery 
test. 

Step 3 
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Stmp 4: Feedback 



Tha mastery teat ehould be 
scored and the feedback provided 
iimnediately to the student. This 
can be done quite simply by a clerk 
who is on duty in the room with the 
equipment. Each item in the mastery 
test should be keyed to some 
available source so that the student 



FEEDBACK 
ON TEST 

(with refer- 
ences to all 
questions) 

Coenponent 4 



Check your 
answers and 
see refer- 
ences for 
questions 
missed. 

Step 4 



can quickly find the answers for questions missed. 



If a student does not reach the prescribed level of 
mastery required he will be asked to go through the unit a 
second time. Past experience has shown that only a small 
proportion of students fail to reach mastery on the first 
try, and of those who do not, all reach mastery after two 
efforts. 

Here are two examples of typical frames (the handout, 
slides, and recorder operating simultaneously) : 



EXAMPLE X 



HANDOUT: 



... a pr.rcentile is a way 
of con^aring your performance to that 
of others. 



SLIDE: 




TAPE: 



Now that's the key to interpreting a percentile 
rank, namely, "conqpared to lAat?" The raw 
score, the number you get right on a test doesn't 
change, but the reported score, the percentile 
rank reported, changes %rtienever the comparison 
group changes. It*s like the fellow who says, 
**How*s your wife?** and you respond, ""Compared 
to what?** It*s the conqparison that's important. 
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EXAMPLE 2 



HANDOUT: 



A normative test is based 
on a comparison of each 
person's score to the scores 
of a pre-selected and pre- 
tested norm group • 



SLIDE: 



NORMATIVE MEASURE 

1. A general area is defined. 

Items are written and tried out. 
Items which discriminat e are maintained. 



A representative group of people is 
tested (the norm group) . 

You are tested. Your score is reported 
a cong>arison to the norm group. 



TAPE: 
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Now here's how a norm- referenced test is built. 
First a general area is defined and the iteais are 
written and tried out. Any item which discriminates 
among the people in the tryout is maintained. 
Remember what we said about discrimination; in other 
words, the good people answer one way and the not 
so good another. Fourth, a representative group 
of people is tested. This is called the norm group 
and it is supposed to be representative of the 
other people who will later take the test. And as 
you can see there, niimber 5, sane other person is 
tested — you and your score is then compared to 
the norm group. It's a comparison: we take your 
test score and we don't report to you your raw score; 
that's not reported. What is important and what is 
reported is how you compare to the norm group. 



-9- 



III. Here are a few situations in which the program might be 
quite useful. 

a. At the funding agency level to help the staff decide 
which among prospective proposals should be funded. A wide 
variety of state, federal, local, and private agencies are in 
the proposal-funding business. Since evaluation is such a key 
determinant in the success of almost any project, it seems 
reasonable that the more complete is the evaluation training of 
the staff, the higher the probability that the staff will be 
able to separate the promising proposals from the less promising. 

b. At the funding agency level to help with the evaluation 
of projects which do get funded. Here the funding agency would 
have the self-instructional units available for use by those 
funded and by agency staff to assist with the evaluation of the 
variety of projects. 

c. At some central depository of audio-visual and self- 
instructional materials for laon to people on projects of all 
kinds. A large school district, a consortium of small districts, 
a college or univeristy, a county board of education, or a 
variety of other agencies may have some central library which 
includes self-instructional materials. Through such a center, 
the units could be made available to anyone with an evaluation 
need . 

d. As part of a foundations program for certain graduate 
school departments. Many graduate programs require "foundations" 
or "tool" courses as a requirement tor an advanced degree. Ex- 
amples include foreign language, computer, or research courses. 
For some doctoral areas, these units seem like a very reasonable 
and relevant substitution. Since the units are completely self- 
instructional, they have the advantage of not requiring the 
university to maintain a faculty member for this part of the 
instructional program. 

e. As a supplement to or replacement of an introductory 
research and/or evaluation course at the university level. Many 
of the topics covered by the units are routinely covered in 
beginning research courses, with the availability of self-in- 
structional units, the instructor would be freed from covering 
this material — working more closely with the students or dealing 
with special problems. Using the units to supplement such a 1 
course has worked very satisfactorily here at Northwestern University. 



The results of this trial are summarized in the paper "On 
Making a Big Lecture Section a 'Good' Course" by John W. Wick. 
A copy of this paper is available on request. The paper will 
appear in Profiles in College Teaching , B.C. Mathis, and W.C. 
McGaghie, editors (Tix press) . 
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f. The entire set of units can be used as a credit- 

?rundi?^.''^''-;^^^!.^°^ in-service teachers. The concept 
xs under consideration at one state university whereby in- 

ItZtttl^Tt^^ ^^^^ ^^«<^its without actually 

^^Jr college to attend classes. In another 

i^t^^' ^^^^^ 2^^^^^ °^ Education is considering estab- 
lishing the set of units as one technique whereby teachers 
can receive salary or promotion credit in the local district 
without actually attending a college. aistrict 

Other uses are certainly possible. These are just a 
few of those which have been suggested at this point. 

IV. RELATED MATERIALS 

Two sets of parallel presentations of this material 
?o?, preparation. These will be available in early 

1973 and will be published by the Charles E. Merrill Pub- 
lishing Company. The first set of parallel materials will 
consist of a consumable student workbook linked to approxi- 
mately six hours of audio instructions. The student work- 
book will co^isist of approximately 350 pages of information, 
worked examples, and problems, which the student will complete 
on his own, following the audio directions given on the 
cassette-type tapes. The coverage of this program will be 
very close to that of the self-instructional slide/tape/printed 
program In addition, Charles E. Merrill Publishing Company 
will publish a text paralleling the four measurement-related 
units in the program. The book will, of course, go into much 
more detail than that found in the units. 

The three programs (the slide/ tape/printed program, which 
IS the topic of this mailing; the consumable student workbook/ 
audio tape program; and the in-depth book aspect) are being 
prepared as independent presentations. The three parallel 
programs do not depend upon each other; but they will be sup- 
plementary. For example, either of the self- instructional 
units could be used very effectively with the r-ore in-depth 
presentation in the book. The multiple format approach has 
been devised to provide for the most flexible possible use 
of these materials. 

V. PURCHASE 

See enclosed price list. 
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Price List 

SELF-INSTRUCTIONAL UNITS ON EVALUATION AND RESEARCH 



Complete Set of Eleven Units $260.00 

includes: 11 sets of slides for the 11 units 
11 60-minute tapes for the 11 units 
One copy each of: 

objectives for the 11 units 

handouts and worksheets for the 11 units 

mastery tests for the 11 units 

keyed copy of mastery tests for the 11 units 



user must provide: 

slide projector 

cassette player (preferably equipped with 

headsets) 

reproduction of the printed material for 
student use 



Single Unit (add $3.00 for handling) $26.00 

includes: 1 set of slides 

1 60-minute cassette tape 
One copy each of: 

objectives 

handout and worksheet 
mastery test 
keyed mastery test 



Extra Copies of Printed Material : 

Package of 30 copies of student materials for a single 
unit (includes objectives, handout , and worksheet only 
for one unit) 

$15.00 

Package of 30 copies of mastery tests for a single unit 

$12.00 

For further information , please write: 

Center for the Teaching Professions 
Northwestern University 
Evanstonr Illinois 60201 
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MAKE CHECKS PAYABLE TO; CENTER FOR THE TEACHING PROFESSIONS 



